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Introduction 

(Please provide a high level description of the invention, including the names of the people who contributed to the invention.] 

This invention allows content producers (e.g., Hollywood) to easily enhance their television, movie and 
commercials for broadcast via the use of standard HTML commands. By allowing these producers to ; 
leverage their existing investment in web-based authoring tools for HTML, we can easily allow for the 
development of data-enhanced shows without needing to develop an entirely new infrastructure specific for 
broadcasts. This invention covers not only how the shows are originally authored, but also the controlling 
uifrastmctureforraedeUvery.dbplay^ Some examples of data 

enhanced shows would include: 

• Providing for ancillary statistics during a sports broadcast, allowing the user to find more 
detailed information on a particular team, or player. 

• mteracuvemerchandismgofclothmg,driiuB,fuim 

broadcast (e g., allowing the purchase of Jerry Seinfeld's shirt during his sitcom show). 

• Additional relevant information on the current topic of a documentary, allowing the user to 
"read more about it" either during or after the broadcast 

One of the key aspects of this invention is allowing the author to treat a streaming motion picture show as 
the "background" in an HTML frame, thereby allowing any HTML commands or extensions to be placed . 
on top of the motion images, in much the same way that existing web browsers allow the author to create a 
static background bitmap which HTML commands can then be placed on top. 

Note that this invention is applicable not only to broadcast sources of streaming analog or digital video 
(satellite, terrestrial television, cable, Internet multicast*), but also to other non-toiditional distribution 
sources for movies (DVD, CDROM, etc). Pre-authored data enhancements could either be dehvered^ 
simultaneously with the video (as in a broadcast source), pre-downloaded and pre-cached on the user s 
hard disk in the middle of the night, or extracted in real-time from a DVD or CDROM. This invention will 
work equally well in all of these cases. 

The inventors are George Moore and Laura Butler. 

Mntoi^b^i^ vhat "new" things your invention does and a high-level description of how it does them).] 

During the interactive television trials by a variety of companies throughout the early 1 990s, one of the 
fundamental problems for creation of enhanced broadcasts ™ <^7™* !° ^^if 

enhanced content in the first place. A variety of companies (Microsoft mcluded) created some basic 
authoring tools, but because this was a new platform, with a different set of APIs, tiiere weren't a lot of 
people creating authoring tools for these environments. Therefore, content authonng for this environment 
was difficult 

With the raise in popularity of the web, and the rapid progress in the capabilities of web browsers, allowing 
third parties to use their existing web-based tools to create enhanced content for television shows means 



PAGE: 1 



that we can instantly draw upon the richness of thousands of tools, plug-ins and controls available for 
modern browsers. This is the main new part of our invention: treating a motion picture as a background, 
surface within a web browser. The other main new part of this invention deals with the header information 
in each of the HTML files which control where the files are stored on the local hard disk, when they are 
displafedrifhchronized with the appropriate scene on the show), how long they are displayed, and 
localization/platform information. This way several sets of localized enhancements could be broadcast in a 
market such as Europe, and each PC will pick out and display the appropriate language enhancements for 
their particular country. 

^SlZpl^Ld^^n^ion of the invention, including the architecture and design details of the implementation. The 

SnZoation. The use of a specific example, shoving ho* the invention sohes the problem bemg ^dressed can be pamcularty 
nlulLYoushouldaUomLtionJhetheryouhavethough^ * 

description may greatly enhance \the efficiency of the process.] 



We combine streaming video in a window, color keying, and an HTML command frame into one : new 
invention. By placing an HTML command frame (such as from the new Microsoft Internet Explorer ^3.0) 
behind a viewport window that is displaying streaming motion video, and then programmmg the on-board 
color keying hardware to exclude all colors except, for example magenta, any non-magenta colored HTML 
commandJkt are men issued wiU^^^ \ 
illustratioTshows this more clearly (this may not print out too well - it is more legible on the screen): 
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Top View (what the user will see) 
Window Frame © HTML Text © 

^ Z 




Streaming Video © 



Edge View (what the programmer will see) 



HTML Command Frame (3) 



Streaming Video Viewport © 



Window Frame ® 



The HTML Command Frame (item #3) is placed behind the Streaming Video Viewport (item #2). If the 
color keying hardware has been programmed to allow any color except magenta to pass through, Ihen any 
non-magenta colored text or graphics displayed in item #3 will be placed on top of the video. In the 
example above, the text "Enhanced Content Text" was issued as an HTML command in the HTML 
Command Frame, and it appears on top of the Porsche 91 1 shown in the Streaming Video Viewport The 
HTML Command Frame (#3) will take care of parsing the web page(s) and associated pieces (animations, 
sounds, bitmaps, ActiveX controls, etc), getting the HTML control to position itself appropriately, 
navigation to the appropriate content page, and color keying as necessary. 

If the user moves or resizes the video window, then all three of these "planes" will be moved together, 
furthering the illusion that it is one object on the screen. Note that this could also run foil-screen without a 
window frame (item #1 would not be drawn), thereby making the PC screen appear to be more like a 
television set than a computer monitor. 



The other key portion of this invention deals with the command and control information for the specific 
HTML Command Frames (#3). This information is stored in an associated control file which tells the 
system when to display and when to tear down specific .HTM files. 

We also create a different default .HTM files for different data back ends, namely: 

• Live (broadcast via satellite or terrestrial television data encoded in the VBI) 

• Recorded (video tape, game cartridge, CD-ROM, DVD disk) 
The timing information necessary for Live vs. Recorded will be different, and can be adjusted by the 
system. These default .HTM files will be also be used in the event there are no enhancements associated 
with a video stream. When this happens you will simply see the streaming video with no extensions or 
enhancements, but the default .HTM files will be "live" so they can wake up and process the commands 
when an enhancement shows up. 

The file format for controlling the display and placement of the associated .HTM files will be something 
like this (we haven't fully fleshed this part out yet): 

<File Header with # of frames, and repeat count> 

<Framesl -N> 

<Header info, with timing start, timing duration, language localization info, eto 
<URL of web page> 

This allows the content author to control when and for how long each HTML "page" will be displayed over 
or along side the video. 

Diagrams and Flow Charts: t , , 

/7b support the description provided above, please include: (a) at least one block diagram showing the architecture of the system 
that implements your Invention, and (b) at least one diagram Illustrating the primary steps performed by your invention.] 

Step 1: During system initialization, the three video planes are created and synchronized with each other. 

The topmost frame (#1) controls the placement and size of the other two planes. 
Step 2- The system tunes to the user selected broadcast channel (satellite, television, cable, Internet 

multicast), or the user inserts a data-enhanced disk (CDROM or DVD). The data enhanced JHTM 

(P^*» conmandTiles are extracted from the appropriate transmission transport and saved on the user's 

local hard disk in a location specified by the show's content author. For each logical grouping of 
.HTM files, there will b e one control file w hich contains the header information shown above. In 
the case where you are dealing with a device with limited memory (e.g., a television set), you may 
not have a local hard disk. When this happens, the data enhancements are simply shown as they 
are broadcast in real-time, with no caching of the enhancements. 
Step 3- In the case of a broadcast which will occur at some point in the future (e.g., tomorrow), an entry is 
placed in the local electronic programming guide database to link the specified enhancements with 
a broadcast network, or an individual show on that network. When the user tunes to that network 
or show at that future time, the corresponding data enhancements are automatically loaded In the 
case of a data-enhanced disk insertion, the corresponding data enhancements are extracted as soon 
as the disk begins playing. . TT ^ # 

Step 4: The system examines the control file to determine at what tune each of the specific .HTM files 
will be shown with the video. This synchronization can either be at a gross level (e.g., +/- 15 
second accuracy as simple offsets from the start of the show as derived from the information about 
the show stored in the local electronic programming guide database), or at a very fine level (e.g., 
frame-level synchronization with the video). 

Additional Information: 



• List the names of any people who contributed to the Invention. 
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